METHOD AND DEVICE FOR GENERATING A PERSON'S PORTRAIT , 
METHOD AND DEVICE FOR COMMUNICATIONS , AND 
COMPUTER PRODUCT 



5 FIELD OF THE INVENTION 

The present invention relates to a technology for 
generating an image of a person' s portrait in which, from 
a two-dimensional image containing a person's face picked 
up by an image sensor, the feature of the subject person 

10 is emphasized to generate the person' s portrait or generate 
a deformed image of the person's portrait. This invention 
also relates to method and device for communication, and 
a computer readable recording medium which stores a computer 
program which when executed on a computer realizes the 

15 methods according to the present invention. 

BACKGROUND OF THE INVENTION 

In recent years, it has become possible to send/receive 
electronic mails not only on a personal computer but also 

20 on a portable terminal. Furthermore, not only documents, 
but also sender' s portrait have been widely attached to the 
electronic mails. When creating such a portrait by using 
a conventional method, it is necessary for the user to 
manually make selections from prepared parts of several tens 

25 or more kinds, such as face outline parts, face parts and 



hairstyles, and combine them all together. In this case, 
a large number of combinations are available, and as a result, 
a great amount of workload is imposed on the user in an attempt 
to create a desired person's portrait. For this reason, 
5 there has been a strong demand for a device which can easily 
and automatically generate a per s on f s portrait using an image 
of that person acquired by a camera. In order to create 
a person's portrait, such a device needs to automatically 
carry out a sequence of processes of extracting the face 
10 portion from the image, detecting eyes, mouth, etc., and 
replacing the desired parts with already prepared desired 
templates . 

As one example of the related technique, Japanese 
Laid-open Patent Application No. 255017/1998 (Tokukaihei 

15 10-255017) discloses a method and device for generating a 
person' s portrait. In this method, with respect to the face 
area detection, (R, G, B) values of skin color are 
preliminarily set to a specified range, and pixels having 
the corresponding color are marked, and clustered so as to 

20 carry out the detection, while the face parts detection is 
carried out by template matching processes. However, with 
respect to the face area detection, 

(1) the face color is inf luencedby illumination conditions, 
resulting in a difficulty in detecting the face area; in 

25 some cases; 



(2 ) the face color is not located within the above-mentioned 
range of the (R, G, B) values due to differences in human 
races, etc . ; and 

(3) when pixels having a color similar to the face color 
5 exists on the background, it is sometimes difficult to detect 

the face area. 

Moreover, with respect to the face parts position 
detection, conventional problems are that, 

(4) when the number of face-part templates is increased 
10 so as to match various face parts contained in an image, 

the amount of calculations increases due to increased 
collation processes, resulting in degradation in the 
execution speed; and 

(5) a dark shadow tends to appear on the periphery of a 
15 face part in an image depending on illumination conditions, 

resulting in a difficulty in the template matching. 

SUMMARY OF THE INVENTION 

It is an object of this invention to provide a person' s 

20 portrait generation device which picks up a two-dimensional 
image that contains the person' s f ace by using an image sensor, 
analyzes characteristics features in the face from the picked 
image, and automatically generates an image in which 
characteristics features of the face are emphasized. It 

25 is another object of this invention to provide a person's 



portrait generation method for such a device. It is still 
another object of this invention to provide a communication 
terminal which incorporates the person' s portrait 
generation device according to this invention, and provide 
5 a communication method thereof . It is still another ob j ect 
of this invention a computer readable recording medium which 
stores a computer program which when executed on a computer 
realizes the methods according to the present invention. 
According to one aspect of this invention, 

10 two-dimensional images containing a person' s face are picked, 
area corresponding to head of the person is extracted based 
on a differential image of the acquired two-dimensional 
images. After that, position of characteristic features 
of the face of that person are detected from the area 

15 corresponding to head of the person. Then, outline of the 
face is detected by detecting a border between the face and 
the background of the image . Finally, the person' s portrait 
in which the characteristics features of the person' s face 
are emphasized is generated based upon the acquired 

20 two-dimensional images and data corresponding to the area 
corresponding to head, position of characteristic features 
of the face, and the outline of the face. 

According to another aspect of this invention, a 
two-dimensional image containing a person' s face is picked, 

25 area corresponding to head of the person is extracted from 



the acquired two-dimensional image. After that, position 
of characteristic features of the face of that person are 
detected by dividing the head area thus extracted into face 
part areas for respective face parts and finding projection 
5 data of binarized images for the respective face parts areas . 
Then, outline of the face is detected by detecting a border 
between the face and the background of the image. Finally, 
the person' s portrait in which the characteristics features 
of the person 7 s face are emphasized is generated based upon 

10 the acquired two-dimensional image and data corresponding 
to the area corresponding to head, position of characteristic 
features of the face, and the outline of the face. 

According to still another aspect of this invention, 
a two-dimensional image containing a person' s face is picked, 

15 and area corresponding to head of the person is extracted 
from the acquired two-dimensional image. After that, 
position of characteristic features of the face of that 
person are detected from the area corresponding to head of 
the person. Then, outline of the face is detected by 

20 detecting a border between the face and the background of 
the image. Finally, the person's portrait in which the 
characteristics features of the person' s face are emphasized 
is generated based upon the acquired two-dimensional image 
and data corresponding to the area corresponding to head, 

25 position of characteristic features of the face, and the 



outline of the face.* 

According to still another aspect of this invention,, 
a two-dimensional image containing aperson' s face is picked, 
and area corresponding to head of the person is extracted 
5 from the acquired two-dimensional image. After that, 
position of characteristic features of the face for each 
of the face parts of that person are detected from the area 
corresponding to head of the person. Then, outline of the 
face is detected by detecting a border between the face and 

10 the background of the image . Finally, the person' s portrait 
in which the characteristics features of the person' s face 
are emphasized by changing the two-dimensional image with 
respect to each of the face parts. 

Furthermore, a communication method and device 

15 according to still another aspect of this invention is 
capable of electronically transmitting or receiving the 
generated portrait using a communication unit. The 
communication device is, for, example, a cellular telephone 
or a PDA. 

20 Furthermore, a computer readable recording medium 

according to still another aspect of the present invention 
stores a computer program which when executed realizes the 
methods according to the present invention. 

Other objects and features of this invention will 

25 become apparent from the following description with 



reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram that shows an arrangement 
5 of a person' s portrait generation device in accordance with 
the first embodiment of the present invention. 

Fig. 2 is an explanatory drawing that shows the 
operation of a head area extracting section in accordance 
with the first embodiment of the present invention. 
10 Fig. 3 is an explanatory drawing that shows a median 

filter in accordance with the first embodiment of the present 
invention . 

Fig. 4 is an explanatory drawing that shows a smoothing 
filter in accordance with the first embodiment of the present 
15 invention. 

Fig. 5 is an explanatory drawing that shows the 
operation of a feature detection section in accordance with 
the first embodiment of the present invention. 

Fig. 6 is an explanatory drawing that shows the 
20 operation of a face outline determining section in accordance 
with the first embodiment of the present invention. 

Fig. 7 is a drawing that shows a specific example for 
a face portrait image that is formed based upon an input 
image by an image processing section in accordance with the 
25 first embodiment of the present invention. 

7 



Fig . 8 is a flow chart indicating the steps in a person' s 
portrait generation method in accordance with the first 
embodiment of the present invention. 

Fig. 9 is a flow chart indicating an image input step 
5 in accordance with the first embodiment of the present 
invention . 

Fig. 10 is a flow chart indicating a head area 
extracting step in accordance with the first embodiment of 
the present invention. 
10 Fig . 11 is a flow chart indicating a feature extracting 

step in accordance with the first embodiment of the present 
invention . 

Fig. 12 is a flow chart indicating a face outline 
determining step in accordance with the first embodiment 
15 of the present invention. 

Fig. 13 is a flow chart indicating an image processing 
step in accordance with the first embodiment of the present 
invention . 

Fig. 14 is an explanatory drawing that shows a specific 
20 example for a face portrait image that is formed based upon 
an input image by an image processing section and a deforming 
method from the original image, in accordance with the second 
embodiment of the present invention. 

Fig . 15 is an explanatory drawing that shows a specific 
25 example for a face portrait image that is formed based upon 



an input image by an image processing section and a deforming 
method from the original image, in accordance with the third 
embodiment of the present invention. 

Fig. 16 is an explanatory drawing that shows a 
communication terminal and a communication method in 
accordance with the fourth embodiment of the present 
invention . 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

A first embodiment of the present invention will be 
explained first with reference to the accompanying drawings . 
Fig. 1 is a block diagram that shows a construction of an 
image generation device in accordance with the first 
embodiment of the present invention. This device comprises 
five sections. That is, an image input section 1, a head 
area extracting section 2, a feature detection section 3, 
a face outline determining section 4 and an image processing 
section 5. 

For convenience of explanation, it will be assumed 
here that only one person is present in an image that has 
been picked up by an image sensor, and that although there 
is change in the position of the person there is no movement 
in the background of the image. 

The image input section 1 including an image sensor 
picks up two-dimensional images. The head area extracting 



section 2 analyzes a plurality of the two-dimensional images 
picked up by the image input section 1 so that an area including 
the head of a person is extracted. The feature detection 
section 3 analyzes the luminance values of the input images 
5 with respect to the area extractedby the head area extracting 
section 2 so that the positions of face parts such as the 
eyes and mouth are detected. Based upon the face parts 
positions detected by the feature detection section 3, the 
face outline determining section 4 analyzes the color of 

10 a specific target area, such as the face color of the person, 
and based upon this analysis, determines the border between 
the target area and the background. Based upon output 
information from the head area extracting section 2, the 
feature detection section 3 and the face outline determining 

15 section 4, the image processing section 5 creates an image 
in which the characteristic features of the target area is 
emphasized from the input images. 

Referring to Fig , 2A to Fig. 2K, a detailed explanation 
will be given of the operation of the head area extracting 

20 section 2 . In the first embodiment of the present invention, 
the head area extracting section 2 calculates a differential 
image by using two images that have time-wise differences 
among the input images picked up by the image input section. 
In this case, in general, it is preferable to use two images 

25 that virtually continue in terms of time. Moreover, in the 



differential image, pixel data are constituted by values 
obtained by binarizing the absolute value of a difference 
between corresponding pixel values. 

Fig. 2A to Fig. 2K are explanatory draws that show 
5 how the head area of a person is detected by using the 
differential image calculated from the images picked up by 
the image input section 1. Fig. 2A shows an image including 
the head of a person picked up by the image input section 
1. Fig. 2B is a differential image that has been binarized. 

10 In other words, a black pixel indicates a pixel in which 
there is any change in the pixel value, that is, a pixel 
that has been moved . In this example, only one person moves ; 
however, black pixels are sparsely mixed into the background 
due to noise contained in the image. Moreover, when any 

15 moving object is contained in the background, black pixels 
are also mixed into the background portion; however, these 
black pixels are processed as noise when the main portion 
of the image is occupied by one person. 

Next, from Fig. 2B, black pixels are retrieved in the 

20 horizontal direction of the image. In other words, with 
respect to black pixels that are aligned along one horizontal 
line, the position of a black pixel is retrieved rightwards 
from the left end, or leftwards from the right end, and the 
position of the black pixel that has been first encountered 

25 is recorded. This process is repeated with respect to all 



the horizontal lines. Fig. 2C shows the results 
(hereinafter, referred to as left side profile) of the 
retrieval rightwards from the left end in which the positions 
of the black pixels that have been first encountered are 
5 connected from the upper end to the lower end. Fig . 2D shows 
the results (hereinafter, referred to as right side prof ile) 
of the retrieval leftwards from the right end obtained in 
the same manner with respect to the positions of the black 
pixels . 

10 As can be seen in these cases, each of the two profiles 

has an irregular edge shape due to the black pixels that 
are noise components sparsely contained in the background. 
Therefore, prior to obtaining the outline of the head area, 
it is necessary to eliminate the irregular edge shape that 

15 forms noise components. 

In order to eliminate this irregular shape, a filtering 
process is preferably carried out. For example, a median 
filter in which the horizontal position in the right and 
left profile is represented as a length value from the end 

20 is applied so as to eliminate the noise components. 

Referring to Fig. 3, a brief explanation will be given 
of the median filter (center value filter) . 

In the median filter, in general, n-number of data 
d[0], . . ., d[n - 1] are sorted in the order to sizes, and 

25 d[k] (0 ^ k ^ n - 1), which is located in the middle in 



the order to sizes, is selected as an output value. 

In Fig. 3, for example, with respect to N-number of 
column data d(l) to d(N), with respect to arbitrary data 
d(k) , a total of (2m + 1) number of data including m-number 
5 of proximity data before and after thereof are sorted in 
the order to sizes, and d(k) is replaced by a value located 
in just the middle number (m + 1) ; thus, the filtering process 
is realized. In this manner, the slope of data (contrast) 
is properly maintained. In other words, as shown in Fig. 

10 3, when high values are continuously inputted as in the case 
of an (x) data group (from the fifth data to the tenth data 
from the top) before the application of the process, the 
contrast (gaps between the- values) is appropriately 
maintained as shown in the Figure indicating a state after 

15 the application of the process. 

In contrast, even when a high value is singly inputted 
as in the case of a (y) data group (from the fifth data from 
the bottom to the third data from the bottom) before the 
application of the process, this is eliminated by taking 

20 the center value of the proximity data as shown in the Figure 
indicating a state after the application of the process, 
thereby making it possible to eliminate the noise. 

Fig. 2E and Fig. 2F respectively show the results 
obtained by applying the median filter to the profiles of 

25 Fig. 2C and Fig. 2D. However, as shown by the results of 



Fig . 2E and Fig. 2F, even the application of the median filter 
sometimes can not eliminate all the noise components 
depending on the degree of noise mixture. Therefore, the 
application of a smoothing filter to Fig. 2E and Fig. 2F 
5 makes it possible to further eliminate the noise components . 

In the smoothing filter, in general, supposing that 
n-number of data d [0], . . . , d[n - 1] exist, for example, 
with respect to arbitrary profile data d[k], (d[k - 1] + 
d[k] + d[k + l])/3 is calculated, and the average value is 

10 outputted as d[k]. 

Referring to Fig. 4, a brief explanation will be given 
of the smoothing filter. 

In Fig. 4, for example, -in connection with N-number 
of column data d (1) to d (N) , with respect to arbitrary 

15 data d(k), an average value of a total of (2m + 1) number 
of data including m-number of proximity data before and after 
thereof are calculated to find an average value, and d(k) 
is replaced by this value; thus, the filtering process is 
realized. In this manner, the data can be smoothed by using 

20 comparatively less amount of calculations . In other words, 
as shown in Fig . 4 , when high values are continuously inputted 
as in the case of an (x) data group (from the fifth data 
to the tenth data from the top) before the application of 
the process, a smoothing process is available as a whole 

25 although the contrast becomes slightly dull as shown in the 



Figure indicating a state after the application of the 
process . 

Moreover, even when a high value is singly inputted 
as in the case of a (y) data group (from the fifth data from 
5 the bottom to the third data from the bottom) before the 
application of the process, noise is clearly eliminated as 
shown in the Figure indicating a state after the application 
of the process. 

Fig. 2G and Fig. 2H show the results of smoothing 
10 processes carried out on the right and left profiles of Fig. 
2E and Fig. 2F. These show that noise components are 
virtually eliminated so that the right and left outlines 
of the head are obtained. 

Next, an explanation will be given of a method in which 
15 a rectangular area enclosing the head area (hereinafter, 
referred to as a head rectangular shape) is determined by 
utilizing the right and left profiles shown in Fig. 2G and 
Fig. 2H. 

Fig. 21 shows a state in which left side profile 6 
20 of Fig. 2G and right side profile 7 of Fig. 2H are superposed. 
Here, in Fig. 21, retrieval is carried out downward, starting 
from the upper end of the horizontal line, with respect to 
the right and left profiles so as to find the first crossing 
portion, and a horizontal line passing through the first 
25 crossing portion is defined as the upper side of the head 



rectangular shape 8. Next, in order to determine the left 
side of the head rectangular shape 8 , the retrieval is further 
continued downward with respect to the left side profile 
6, A pixel position at which the distance from the right 
5 end of the image first starts to decrease is recorded, and 
a vertical line passing through this point is defined as 
the left side of the head rectangular shape 8, and recorded. 
With respect to the right side profile 7, the retrieval is 
made in the same manner so that a vertical line passing through 

10 the resulting pixel position is defined as the right side 
of the head rectangular shape 8, and recorded. 

With respect to the remaining lower side of the head 
rectangular shape 8, a candidate position is determined by 
utilizing the fact that the ratio of the width and the height 

15 of the head rectangular shape 8 of a general person is 
virtually constant. More specifically, supposing that the 
width is W, the height is H and the constant representing 
the average ratio is k, H/W = k is obtained. Here, the value 
of W is given by the distances of the right side and left 

20 side of the head rectangular shape 8 that have already been 
determined, and the position (the distance from the upper 
end of the image) p of the upper side has already been 
determined; therefore, supposing that the candidate 
position of the lower side to be found is p f , p' (the distance 

25 from the upper end of the image) is calculatedby the following 



expression : 

p' = Jc x W + p. 

Fig. 2J shows a state in which the head rectangular 
shape 8 thus determined from the right and left profiles 
5 is displayed in a superposed manner. Finally, the head 
rectangular shape 8 is determined as a silhouette of a person 
to which movements of the person are reflected as shown in 
Fig. 2K that is obtained in a manner so as to enclose the 
head. 

10 Next, an explanation will be given of the operation 

of the feature detection section 3. Fig. 5A to Fig. 5E is 
a drawing that explains an example in which luminance values 
of an input image within the head rectangular shape 8 are 
analyzed so that the positions of characteristic features 

15 of the eyes, nose, mouth, etc. are specified. 

Fig. 5A shows an example of an input image. Fig. 5B 
shows a state in which the head rectangular shape 8 extracted 
by the head area extracting section 2 is displayed in a 
superposed manner. Fig. 5C shows areas 9 to 17 used for 

20 specifying rectangular areas that are analyzing ranges for 
identifying the positions of respective face parts that form 
characteristic features, and these areas are obtained by 
preliminarily finding relative positions of the respective 
face parts with respect to the head shape of a general person . 

25 In other words , with respect to the upper , lower, right 



and left sides of each of the rectangular shape areas, the 
right eye is specified by points 14, 15, 10 and 9, the right 
eye is specified by points 14, 15, 9 and 13, the nose is 
specified by 15, 16, 11 and 12, and the mouth is specified 
5 by 16, 17, 11 and 12. 

Next, among the face parts detections by the use of 
Fig, 5D, for example, an explanation will be given of a method 
by which the position of the right eye is detected. The 
detection area surrounded by 14, 15, 9 and 10 is obtained 

10 by binarizing pixel values of the input image, and with 
respect to the pixel values, values greater than a threshold 
value are represented by 0 (while pixels) and values other 
than this case are represented by 1 (black pixels) . In order 
to calculate the detection position (the center of the pupil) 

15 of the right eye by using the center-of -gravity position, 
projection data are obtained by adding values of respective 
pixels in the binarized image within the detection area for 
each row and each column, and the center-of -gravity position 
of each of these is calculated so that the horizontal position 

20 and vertical position of the right eye detection position 
are obtained. In these cases, 25 and 26 respectively 
represent projection data in the vertical direction (for 
each column) and pro j ection data in the horizontal direction 
(for each row). The center-of-gravity positions are 

25 calculated from these projection data. In the horizontal 



direction, 27 is obtained as the center-of -gravity position, 
and in the vertical direction, 28 is obtained as such, with 
the result that the right-eye detection position 19 shown 
in Fig. 5E is obtained. 
5 Here, with respect to the horizontal and vertical 

positions, the positions, each providing the maximum value 
of the projection data, may be used instead of the 
center-of -gravity of the projection data so as to calculate 
these positions. 

10 In the same manner, the detection position 18, 20 or 

21 shown in Fig. 5E for each of the left eye, nose and mouth 
is found by calculating the center of gravity from the 
binarized image within each of the rectangular shape areas. 
The above explanation has been given of the eyes, nose 

15 and mouth serving as the characteristic features; however, 
in an attempt to improve the precision as a person' s portrait, 
the detecting process may be carried out on other parts 
elements, such as the eyebrows, front hairstyle, cheeks and 
ears . 

20 Next, referring to Fig. 6A to Fig. 6G, an explanation 

will be given of the operation of the face outline determining 
section 4. In the present embodiment, the face outline 
determining section 4 calculates the RGB value of the average 
color of pixels in the proximity of each face parts position 

25 by using information of the face parts position detected 



by the head feature detection section 3 based upon the input 
image of the image input section 1, and converts this value 
to a value on the HSV color system; thus, the RGB values 
of all the pixels of the input image are converted to the 
HSV values, and these values are plotted on the HSV color 
system. At this time, an area centered on the HSV value 
of the average color is set on the HSV color system map, 
and the pixel data located within the area is again returned 
to the RGB color system, and plotted on the original input 
image. Based upon the results of these processes, the face 
area is estimated. 

In this case, with respect to the conversion of the 
color system from the RGB values to the HSV values , for example , 
a method disclosed in US, Patent No. 5586197 is used so as 
to realize the conversion. Here, the RGB color system uses 
a cubic shape having R, G, B as its axes, while the HSV color 
system uses a cylinder shape as shown in Fig. 6A. With 
respect to the HSV values, the value V (luminance) is 
represented by the height from the bottom of the cylinder 
serving as the reference as shown in Fig. 1A, the value S 
(the degree of saturation, the density of color) is 
represented by the distance from the center axis C of the 
cylinder as shown in Fig. 6B, andH (the degree of chrominance, 
the kind of color) is represented by an angle from a line 
CR of a mapped red color with the values of G and B of the 

20 



RGB values being set to 0 on a disc that is a cross-section 
of the cylinder. 

Fig. 6C shows an example in which upon conversion of 
the RGB values of an input image into HSV values , these values 
5 are plotted on the disc shown in Fig. 6B with differences 
in the value of V being ignored. 

Next, an explanation will be given of a system in which 
an area is provided on the HSV color system based upon the 
average color in the vicinity of the face parts position 

10 and the face area is specified based upon the color within 
the area. Fig. 6D and Fig. 6E show color judgment areas 
for extracting the face area, and 22 represents a position 
obtained when the average color (RGB value) in the vicinity 
of the face parts position is converted into an HSV value. 

15 As shown in Fig. 6D, the shape of the color judgment area 
may be set as an elliptical shape area 23 with a line CX 
serving as a major axis, when a crossing point of the 
circumference and the extended line of the line connecting 
the average color position 22 and the center C is set as 

20 X, or as shown in Fig. 6E, a delta shape area 24 may be set. 
In either of the cases, pixels within the area including 
the average color position 22 and having colors similar to 
this can be specified as face area candidates. 

An explanation will be given of a system in which, 

25 by analyzing the positions of the pixels serving as face 



area candidates that have beendetermined as described above , 
the face outline, especially, the border line in the vicinity 
of the jaw, is estimated- The determining process of the 
face outline in the vicinity of the jaw is important in 
5 creating a person 7 s portrait. In other words, the case in 
which the right and left smoothed profiles 6 and 7 are simply 
used as the face outline in the vicinity of the jaw fails 
to provide a person' s portrait sufficiently representing 
the characteristic features of the face, and in order to 
10 obtain a superior person's portrait, it is necessary to 
individually determine the face outlines in the vicinity 
of the jaw. 

First, with respect to a head outline 29 shown in Fig. 
6F, the inside of the head rectangular shape 8 is analyzed 

15 by the above-mentioned specifying method of the face area 
by using the differential image used in the area extracting 
section 2, partial elliptical shapes of four corners are 
calculated and set in amanner so as to surround this specified 
face portion, and one portion of each of the right and left 

20 sides of the head rectangular shape 8 is further added to 
the partial elliptical shape so that the outline of the face 
including a jaw border line 30 is formed. 

More specifically, on the lower side surrounded by 
the right and left side of the head rectangular shape 8, 

25 that is, within the semi-elliptical area of the head outline 



29 , the rate of the face area candidate pixels, explained 
before, is calculated. In this case, the semi-elliptical 
shapes of the head outline 29 are contracted toward the 
respective centers at the same curvature until the calculated 
5 rate has exceeded a predetermined threshold value, thereby 
determining the jaw border line 30 . Fig . 6G shows an example 
of the estimated results of the jaw border line 30 obtained 
by applying the above-mentioned technique to an actual image . 

Lastly, an explanation will be given of the operation 

10 of the image processing section 5 . In the first embodiment, 
the image processing section 5 generates a person' s portrait 
of a face image contained in an original image by using the 
head rectangular shape 8 detected by the head area extracting 
section 2, the positions of the eyes 18, 19 and the mouth 

15 position 21 detected by the feature detection section 3, 
the head outline 29 detected by the face outline determining 
section 4 and the estimated jaw border line 30 . Fig. 7 shows 
an example of a person' s portrait image generated by the 
image processing section 5. In this image, the upper half 

20 of the head outline 29 and the portion surrounded by the 
jaw border line 30 thereof are extracted from the input image 
to form a face image, and after having been subjected to 
an edge-emphasizing process, this is binarized to form a 
line image. At this time, the inside of the face area may 

25 be painted using the average color to which the face color 

23 



calculated by the face outline determining section 4 is 
reflected; thus, it is possible to obtain the person's 
portrait image as a color image. 

The above-mentioned description has discussed the 
5 construction of the person's portrait generation device, 
and the following description will briefly discuss the 
person' s portrait generation method by reference to a flow 
chart that summarizes its sequences of processes. 

Fig, 8 is a flow chart that shows the person' s portrait 

10 generation method of the present invention corresponding 
to Fig. 1. Steps 101 to 105 shown in this diagram are 
performed respectively by the sections 1 to 5 shown in Fig. 
1. It is to be noted that the sequence of steps 103 and 
104 may be reversed. 

15 Fig. 9 to Fig. 12 are flow charts showing detail 

processing in the steps 101 to 104 shown in Fig. 8. 

In the image input step 101 shown in Fig. 9, first, 
two image buffers in which input images are stored are 
initialized. Next, image input is started. Then, image 

20 data are respectively stored in the two image buffers, and 
if no other pickup image exists, the image pickup process 
is completed, and the image data are outputted. 

In a head area extracting step 102 shown in Fig. 10, 
first, based upon the image data outputted from the two image 

25 buffers, calculations are made to generate a differential 



image between binarized frames. Next, the differential 
image between the binarized frames is analyzed to extract 
a head rectangular shape. 

In a feature detection step 103 shown in Fig. 11, first, 
5 based upon the rectangular area extracted by the head area 
extracting step 102 and relative positions of the respective 
face parts with respect to the head shape that have been 
preliminarily provided, an analyzing range used for 
detecting a position of each of the face parts is determined . 

10 Next, the image within the analyzing range is analyzed for 
each of the face parts so that the center position of each 
of the face parts is detected. 

In the face outline determining step 104 shown in Fig. 
12, first, based upon the image inputted from the image input 

15 step 101, the average color of the face is calculated by 
using RGB pixel data in the vicinity of each of the face 
part positions detected by the feature detection step 103. 
Then, within the rectangular area extracted in the head area 
extracting step 102, pixels having RGB values close to the 

20 average color of the face calculated in the first step in 
a specified range are recorded as skin-color pixels . Next, 
the skin- color pixels thus recorded are analyzed to determine 
the face outline. 

The above-mentioned image generation calculating 

25 method from the picked-up image is realized by an image 



generation program, and the program is recorded in a 
recording medium that can be read by a computer, etc., and 
supplied . 

The first embodiment has exemplified the case in which, 
in the head area extracting section 2, in order to obtain 
a differential image by using two time-wise consecutive 
images among input images picked up the image input section 
1, the absolute value of the difference of the corresponding 
pixels is binarized. However, a method other than this, 
for example, a method .in which an image is subjected to a 
process for extracting the outline of a person, such as an 
edge analysis, so as to extract the outline including the 
head and shoulder as described in the first embodiment, may 
be used, and thereafter, in the same manner as described 
above, a person's portrait may be generated. 

Moreover, the first embodiment has exemplified the 
case in which, in the head area extracting section 2, a 
differential image is calculated by using two time-wise 
consecutive images among input images picked up the image 
input section 1. However, for example, an image 
corresponding to one frame picked up by the image input 
sectionl is used, and supposing that only one person is 
included in the image and that the background has a uniform 
color, a new image is formed by shifting this image up and 
down or right and left by several pixels, and to these two 



images, the same process as the differential image process 
of the head area extracting section 2 described in the first 
embodiment may be applied; thus, it is possible to obtain 
the same results. 

As described above, in the first embodiment, based 
upon two- dimensional images picked up by the image input 
section 1, a characteristic feature from the person's face 
is detected by analyzing the luminance of pixel values, and 
the person' s portrait, etc. , that is colored based upon. the 
calculated average color in the vicinity of the 
characteristic feature is formed, that is, an image 
emphasizing the characteristic feature of the pickup subject 
is created. Therefore, without the need of template 
matching, the characteristic feature of the face parts, etc. 
are automatically detected, and the skin color, etc. of the 
face is calculated from the picked-up image . Consequently, 
it is not necessary to specify the color beforehand, and 
with respect to the background, in the case of a virtually 
stationary background, the pickup sub j ect, such as a person, 
can be detected stably independent of the kinds of the 
background, with the result that an image in which the 
characteristic feature of the pickup subject are emphasized, 
such as a person' s portrait formed by the results of picked-up 
images, can be generated in a stable manner. 

A second embodiment of the present invention will be 



explained next. The arrangement of a person's portrait 
generation device in accordance with the second embodiment 
is the same as that of the first embodiment. In the second 
embodiment, in addition to the functions explained in the 
5 first embodiment, in the image processing section 5, the 
results of the detection of each of the face parts positions, 
such as the eyes, the mouth, etc., made by the feature 
detection section 3 are utilized so as to deform the person' s 
portrait derived from a face image, thereby forming a 

10 deformed face image representing emotional changes, such 
as a smiling face and a crying face. 

Fig. 14A to Fig. 14F are examples that show such 
deformations of a person's portrait. By utilizing a 
portrait shown in Fig. 14A (which is the same as the portrait 

15 shown in Fig. 7), the person's portrait is deformed based 
upon the positions of the eyes and mouth, thereby obtaining 
the results shown in Fig. 14BtoFig. 14F. These are examples 
of the person's portrait deformations. Fig. 14B shows a 
smiling face, Fig. 14C shows a crying face, Fig. 14D shows 

20 an angry face, Fig. 14E shows a fat face, and Fig. 14F thin 
face . 

Next, an explanation will be given of specific 
sequences of the face deformations. First, the face 
deformations shown in Fig. 14B to Fig. 14D are obtained by 
25 deforming a person' s portrait image in accordance with 



directions of arrows determined based upon the positions 
of the eyes and the position of the mouth shown in Fig. 14G 
to Fig. 141 respectively- Moreover, the face deformations 
shown in Fig. 14E and Fig. 14F are obtained by deforming 
5 the person's portrait image so as to expand and shrink in 
the directions of arrows determined based upon the j aw line , 
as shown in Fig. 14J to Fig. 14K. 

As described above, a person's portrait is generated 
based upon an input image, and it is deformed based upon 

10 the detection positions of the face parts, thereby forming 
the person' s portraits representing emotional changes such 
as a crying face and a smiling face; thus , it becomes possible 
to automatically generate such images without the necessity 
of template selection by the user. 

15 Next, Fig. 13 is a flow chart showing steps in the 

image processing step 105. 

First, based upon an input image within the face outline 
calculated by the face outline determining step 104, a 
person's portrait corresponding to Fig. 7 is generated. 

20 Next, after discriminations have been made as to whether 
or not an insertion of any of the face parts is required 
or whether or not a deformation of any of the face parts 
is required, a desired person' s portrait image is obtained. 

The above-mentioned image generation calculating 

25 method from the picked-up image is realized by an image 



generation program, and the program is recorded in a 
recording medium that can be read by a computer, etc., and 
supplied . 

In the second embodiment, the explanation has been 
given of the deformations of a person's portrait image by 
the use of deformations of the face parts. However, for 
example, the face parts from an original image may be used, 
and the original image may be partially deformed based upon 
the detection positions of the face parts so as to provide 
face deformations representing various emotions . Moreover, 
in the second embodiment, the explanation has been given 
of the case in which the deformations are made based upon 
the positions of the eyes and the position of the mouth. 
However, characteristic features other than these, for 
example, the positions of the nose and the ears, may be used 
as the references on which the deformations are made. 
Furthermore, in the second embodiment, with respect to the 
deforming method for emotional face images, the explanation 
has been given by reference to Fig . 14GtoFig. 14K. However, 
the present invention is not intended to be limited by this 
method, and any known method may of course be used to provide 
the deformations. 

A third embodiment of the present invention will be 
explained next. The arrangement of a person's portrait 
generation device in accordance with the third embodiment 
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is the same as that of the first embodiment. In the third 
embodiment , in addition to the functions explained in the 
first embodiment, in the image processing section 5, other 
face parts are put at the face parts positions, such as the 
5 eyes, the mouth, etc., detected by the feature detection 
section 3, in a replacing manner, so as to form another face 
image different from the person's portrait image shown in 
the first embodiment. 

Fig. 15A to Fig. 15E are drawings that explain how 

10 already prepared desired face parts are inserted to the 
positions of the eyes and mouth of the person's portrait 
image shown in Fig. 7 to form another face image. Fig. 15A 
shows a case in which asterisk marks are inserted at the 
positions of the eyes andmouth. Further, when the asterisk 

15 mark corresponding to the right eye is replaced with an image 
of a right eye shown in Fig. 15B,the asterisk mark 
corresponding to the left eye is replaced with an image of 
a left eye shown in Fig. 15C, and the asterisk mark 
corresponding to the mouth is replaced with an image of a 

20 mouth shown in Fig. 15D, then the portrait shown in Fig. 
15E is obtained. 

As described above, in a person's portrait generated 
from an input image, other face parts images are inserted 
to the face parts positions automatically detected, thereby 

25 forming another face image. Thus, it is possible to omit 



the user's manually specifying operations related to parts 
positions, and also to automatically generate various face 
images by preparing a number of such face parts. 

A fourth embodiment of the present invention will be 
5 explained next. The fourth embodiment exemplifies a case 
in which the aforementioned' person' s portrait generation 
device or person's portrait generation method is applied 
to a communication terminal. 

Fig. 16 is a drawing that exemplifies a case in which 

10 the present invention is applied to a mobile telephone 31 
serving as a communication terminal. As shown in Fig. 16A, 
a face image is picked up by a small-size camera 32 attached 
to the mobile telephone 31. Then the image is subjected 
to a face parts position detection and a person's portrait 

15 generating process to obtain the person's portrait shown 
in Fig. 16B. This portrait (data of the portrait) may be 
then transmitted to a desired destination. On the other 
hand, the portrait shown in Fig. 16B may be deformed as shown 
in Fig . 16C . The portrait maybe deformed to express person' s 

20 emotions such as cry, smile, anger and joy, or expressing 
a fat face or a thin face, thereby making it possible to 
transmit the resulting person's portrait data. Moreover, 
face parts position information may be detected from the 
port rait shown in Fig . 16B, and the face parts may be exchanged 

25 with other parts or symbols as shown in Fig. 16D. The 



portraits shown in Fig. 16C or Fig. 16D may be transmitted 
to desired destination. 

The following description will propose examples of 
methods for actually using these person' s portraits that 
5 are effectively used at the time of communication, such as 
mail exchange . For example, a setting may be made in which, 
on a waiting screen at the time of mail receiving, the user' s 
portrait is displayed with changes in emotions, such as 
crying or smiling, depending on the senders; thus, it is 

10 possible to immediately judge who sent a mail or what kind 
of person sent a mail. 

In another example of the application, mail senders' 
portraits are preliminarily registered, and a setting is 
made so that, upon receipt of a mail, the corresponding 

15 sender's portrait is displayed; thus, it is possible to 
immediately judge who sent the mail. Moreover, even when 
the receiver is not an acquaintance, sending the sender's 
portrait may make it possible to provide a friendly mood. 
Furthermore, mail subjects may include items such as 

20 "sad information" and "good information", and a deformed 
person' s portrait expressing a crying state, smiling state, 
etc. is provided on the receiver's side; thus, it becomes 
possible to immediately judge the contents of the mail by 
the expression of the portrait. 

25 Here, the person's portrait formed by the present 



invention may be utilized as various indicators for mobile 
communication apparatuses, etc. For example, a setting may 
be made so that, when the battery is running out, the user' s 
thin portrait is displayed; thus, it is possible to easily 
5 know the exchanging time or charging time for the battery. 
Moreover, when unread received mails have been accumulated, 
the receiver' s f at face maybe displayed so that it is possible 
to easily know the degree of accumulatedmails . Furthermore, 
when direct sun light enters the screen, a dazzled face may 

10 be displayed thereon so that the corresponding information 
is easily given. 

With respect to other different applications, when 
a game, etc., is played, upon achievement of a high mark, 
a smiling face may be displayed while upon receipt of a low 

15 mark, a crying face may be displayed; thus, the game becomes 
more enjoyable. Moreover, the receiver's portrait may be 
combined with a transmitted person' s portrait to have fun. 
Furthermore, parts such as beard, wigs, etc. may be 
preliminarily prepared, and these may be combined with a 

20 person's portrait to have fun. Furthermore, an angry face 
maybe colored in red, or tear parts, preliminarily prepared 
separately, may be added to a crying face to have fun. 

The above-mentioned communication terminal is not 
intended to be limited to the mobile telephone, and any 

25 communication terminal may be used as long as it can transmit 



and receive person's portrait images. For example, it can 
be applied to PDAs (Personal Digital Assistant) , notebook 
PCs, television telephones. 

Moreover, in accordance with the person's portrait 
5 generation device of the present invention, person' s 
portraits may be attached to electric mails, etc. used in 
mobile telephones and personal computers. Alternatively 
in a video conference, etc., using cameras between remote 
places, instead of showing real images of the participants' 

10 faces, the participants' portrait images automatically 
generated may be given on the screen, and in response to 
audio data in the discussion, the mouth portions of the 
portraits may be moved; thus, the participants are allowed 
to have realistic feelings, while avoiding a feeling of 

15 oppression the users might have due to the appearance of 
their actual face on the screen. 

Since the present invention is arranged and has 
sequences as described above, it provides the following 
effects : 

20 The person' s portrait generation device in accordance 

with the present invention is provided with the image input 
section, the head area extracting section, the feature 
detection section, the face outline determining section and 
the image processing section; therefore, without the need 

25 of template matching, the characteristic features of the 



face parts, etc. are automatically detected and the skin 
color, etc. of the face is calculated from the picked-up 
image so that it is not necessary to specify the color 
beforehand. Moreover, with respect to the background, in 
5 the case of a virtually stationary background, the pickup 
subject, such as a person, can be detected stably independent 
of the kinds of the background, with the result that an image 
in which the characteristic features of the pickup subject 
are emphasized, such as a person's portrait formed by the 
10 results of picked-up images, can be generated in a stable 
manner . 

Moreover, the communication terminal of the present 
invention is provided with the image input section, the head 
area extracting section, the feature detection section, the 

15 face outline determining section and the image processing 
section, and the person' s portrait thus f ormedis transmitted 
and received through the communication unit; therefore, 
without the need of template matching, the characteristic 
features of the face parts, etc. are automatically detected 

20 and the skin color, etc. of the face is calculated from the 
picked-up image so that it is not necessary to specify the 
color beforehand. Moreover, with respect to the background, 
in the case of a virtually stationary background, the pickup 
subject, such as a person, can be detected stably independent 

25 of the kinds of the background, with the result that an image 



in which the characteristic features of the pickup subject 
are emphasized, such as a person's portrait formed by the 
results of picked-up images, can be generated in a stable 
manner, and the person's portrait thus formed is utilized 
so as to inform the user of the transmission and receipt 
and is also utilized as an indicator for the communication 
terminal so that it becomes possible to carry out pleasant 
communications while having fun. 

The person' s portrait generation method of the present 
invention is provided with the image input step, the head 
area extracting step, the feature detection sequence, the 
face outline determining sequence and the image processing 
sequence; therefore, without the need of template matching, 
the characteristic features of the face parts, etc. are 
automatically detected and the skin color, etc. of the face 
is calculated from the picked-up image so that it is not 
necessary to specify the color beforehand. Moreover, with 
respect to the background, in the case of a virtually 
stationary background, the pickup sub j ect, such as a person, 
can be detected stably independent of the kinds of the 
background, with the result that an image in which the 
characteristic features of the pickup subject are emphasized, 
such as a person' s portrait formed by the results of picked-up 
images, can be generated in a stable manner. 

Moreover, the communication method of the present 
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invention is provided with the image input step, the head 
area extracting step, the feature detection step, the face 
outline determining step, the image processing step and the 
step which, upon receipt of information, displays the 
5 person's portrait thus formed so as to inform the user of 
the receipt of the information; therefore, without the need 
of template matching, the characteristic features of the 
face parts, etc. are automatically detected and the skin 
color, etc. of the face is calculated from the picked-up 

10 image so that it is not necessary to specify the color 
beforehand. Moreover, with respect to the background, in 
the case of a virtually stationary background, the pickup 
subject, such as a person, can be detected stably independent 
of the kinds of the background, with the result that an image 

15 in which the characteristic features of the pickup subject 
are emphasized, such as a person's portrait formed by the 
results of picked-up images, can be generated in a stable 
manner, and the person' s portrait thus formed is utilized 
so as to inform the user of the transmission and receipt 

20 so that it becomes possible to carry out pleasant 
communications while having fun. 

Furthermore, the recording medium of the present 
invention in which the person' s portrait generation program 
is recorded is provided with: the head area extracting 

25 program for extracting a head area of a person from a 



differential image of a plurality of two-dimensional images 
picked up by an image input section, the feature detection 
program for detecting positions of characteristic features 
of the person' s portrait by dividing the head area into face 
part areas for respective face parts, finding projection 
data of binarized images for the respective face parts areas 
and detecting the position of each of the face parts based 
upon a center-of -gravity position of the projection data, 
the face outline determiningprogramf or determining a border 
between a face outline and a background within the head area 
by specifying a skin color area of the two-dimensional image, 
and the image processing program for creating a person's 
portrait in which the characteristics in the person' s face 
is emphasized by changing the two-dimensional image with 
respect to each of the face parts; therefore, the person's 
portrait thus generated is read by a communication terminal, 
etc. capable of transmitting and receiving images so that 
it is utilized so as to inform the user of the transmission 
and receipt, and is also utilized as an indicator for the 
communication terminal . 

Although the invention has been described with respect 
to a specific embodiment for a complete and clear disclosure, 
the appended claims are not to be thus limited but are to 
be construed as embodying all modifications and alternative 
constructions that may occur to one skilled in the art which 



fairly fall within the basic teaching herein set forth. 
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